{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LoRA Serving"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RTP-LLM supports **Static LoRA** and **Dynamic LoRA** inference modes. For information on LoRA principles, refer to [Hugging Face Official Documentation](https://huggingface.co/docs/peft/conceptual_guides/lora)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serving Single Adaptor（Static LoRA）\n",
    "| Feature | Description |\n",
    "|------|------|\n",
    "| **Fusion Method** | Before inference, the base model and specified LoRA weights are **permanently fused** (irreversible). |\n",
    "| **Applicable Scenarios** | Scenarios requiring **single LoRA** with pursuit of optimal performance. |\n",
    "| **Limitations** | After fusion, the base model cannot be restored; the original output before applying LoRA cannot be obtained simultaneously. |\n",
    "\n",
    "### Usage\n",
    "- When the startup parameter `--lora_info` contains only **1 element**, it automatically enters static mode.\n",
    "- Example: ```--lora_info {\"test0\":\"/mnt/nas1/lora/taoshang_qwen_lora_18000/lora\"}```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wait_sever_done(server_process, port: int, timeout: int = 1600):\n",
    "    host = \"localhost\"\n",
    "    retry_interval = 1  # Retry interval (seconds)\n",
    "    start_time = time.time()\n",
    "\n",
    "    port = str(port)\n",
    "\n",
    "    logging.info(f\"Waiting for pid[{server_process.pid}] to start...\\nPort {port}\")\n",
    "    while True:\n",
    "        try:\n",
    "            # Try to connect to the specified host and port\n",
    "            sock = socket.create_connection((host, port), timeout=timeout)\n",
    "            sock.close()\n",
    "            logging.info(f\"Port {port} started successfully\")\n",
    "            return True\n",
    "        except (socket.error, ConnectionRefusedError):\n",
    "            # If connection fails, wait for a while before retrying\n",
    "            time.sleep(retry_interval)\n",
    "\n",
    "            if (\n",
    "                not psutil.pid_exists(server_process.pid)\n",
    "                or server_process.poll()\n",
    "            ):\n",
    "                logging.warning(\n",
    "                    f\"Process:[{server_process.pid}] status abnormal, service startup failed, please check log files\"\n",
    "                )\n",
    "                return False\n",
    "            # If waiting time exceeds the preset timeout, give up waiting\n",
    "            if time.time() - start_time > timeout:\n",
    "                logging.warning(\n",
    "                    f\"Waiting for port {port} to start timed out, please check log files\"\n",
    "                )\n",
    "                return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "port=8090                \n",
    "server_process = subprocess.Popen(\n",
    "        [\"/opt/conda310/bin/python\", \"-m\", \"rtp_llm.start_server\",\n",
    "         \"--checkpoint_path=/mnt/nas1/hf/models--Qwen--Qwen1.5-0.5B-Chat/snapshots/6114e9c18dac0042fa90925f03b046734369472f/\", \n",
    "         \"--model_type=qwen_2\",\n",
    "         \"--lora_info='{\\\"test1\\\": \\\"/mnt/nas1/lora/qwen_lora_18000/lora\\\"}'\"\n",
    "         f\"--start_port={port}\"\n",
    "         ]\n",
    "    )\n",
    "wait_sever_done(server_process, port)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://localhost:{port}\"\n",
    "json_data = {\n",
    "     \"prompt\": \"who are you\",\n",
    "     \"generate_config\": {\"max_new_tokens\": 32, \"temperature\": 0}\n",
    "}\n",
    "\n",
    "response = requests.post(url, json=json_data)\n",
    "print(f\"Output 0: {response.json()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_server(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serving Multiple Adaptors(Dynamic LoRA)\n",
    "| Feature | Description |\n",
    "|------|------|\n",
    "| **Fusion Method** | Load LoRA dynamically as a plugin during inference, **without modifying base weights**. |\n",
    "| **Applicable Scenarios** | Scenarios requiring **multiple LoRA switching** or **dynamic selection by request**. |\n",
    "| **Limitations** | Only some models support LoRA |\n",
    "\n",
    "### Usage\n",
    "- Specify the LoRA to be used for this request through `generate_config.adapter_name`.\n",
    "- Field rules\n",
    "- Type: `str` or `list[str]`\n",
    "- The number of elements and order must be **completely aligned** with `prompt`\n",
    "- Leave empty or omit to indicate **not using LoRA**\n",
    "- When the startup parameter `--lora_info` contains **multiple** elements, it automatically enters dynamic mode.\n",
    "- Example: ```--lora_info {\"test0\":\"/mnt/nas1/lora/qwen_lora_18000/lora\", \"test1\":\"/mnt/nas1/lora/qwen_lora_18000/lora\"}```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "server_process = subprocess.Popen(\n",
    "        [\"/opt/conda310/bin/python\", \"-m\", \"rtp_llm.start_server\",\n",
    "         \"--checkpoint_path=/mnt/nas1/hf/models--Qwen--Qwen1.5-0.5B-Chat/snapshots/6114e9c18dac0042fa90925f03b046734369472f/\", \n",
    "         \"--model_type=qwen_2\",\n",
    "         \"--lora_info='{\\\"test1\\\": \\\"/mnt/nas1/lora/qwen_lora_18000/lora\\\", \\\"test0\\\": \\\"/mnt/nas1/lora/qwen_lora_18000/lora\\\"}'\",\n",
    "         f\"--start_port={port}\"\n",
    "         ]\n",
    "    )\n",
    "wait_sever_done(server_process, port)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = f\"http://localhost:{port}\"\n",
    "json_data = {\n",
    "    \"prompt_batch\": [\n",
    "        \"<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nhello|im_end|>\\n<|im_start|>assistant\",\n",
    "        \"<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nhello|im_end|>\\n<|im_start|>assistant\",\n",
    "        \"<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nhello|im_end|>\\n<|im_start|>assistant\",\n",
    "        \"<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nhello|im_end|>\\n<|im_start|>assistant\",\n",
    "    ],\n",
    "    \"generate_config\": {\n",
    "        \"max_new_tokens\": 500,\n",
    "        \"top_k\": 1,\n",
    "        \"top_p\": 0,\n",
    "        \"adapter_name\": [\"test0\", \"\", \"test0\", \"\"]\n",
    "    }\n",
    "}\n",
    "response = requests.post(\n",
    "    url + \"/generate\",\n",
    "    json=json_data,\n",
    ")\n",
    "print(f\"Output 0: {response.json()['response_batch'][0]}\")\n",
    "print(f\"Output 1: {response.json()['response_batch'][1]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "stop_server(server_process)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
